Image processing apparatus for superimposing windows displaying video data having different frame rates

ABSTRACT

A method of transferring image data to a composite memory space comprises including masking data defining a reserved output area in a first memory space and containing first time-varying data having a first frame rate associated therewith. Second time-varying image data is stored in a second memory space and is associated with a second frame rate. At least part of the first image data is transferred to the composite memory space and at least part of the second image data is transferred to the composite memory. The mask data is used to provide the at least part of the second image data such that, when output, the at least part of the second image data occupies the reserved output area.

FIELD OF THE INVENTION

This invention relates to a method of transferring image data of thetype, for example, represented by a display device and corresponding totime-varying images of different frame rates. This invention alsorelates to an image processing apparatus of the type, for example, thattransfers image data for representation by a display device andcorresponding to time-varying images of different frame rates.

BACKGROUND OF THE INVENTION

In the field of computing devices, for example portable electronicequipment, it is known to provide a Graphical User Interface (GUI) sothat a user can be provided with output by the portable electronicequipment. The GUI can be an application, for example an applicationknown as “QT” that runs on a Linux™ operating system, or the GUI can bean integral part of an operating system, for example the Windows™operating system produced by Microsoft Corporation.

In some circumstances, the GUI has to be able to display multiplewindows, a first window supporting display of first image data thatrefreshes at a first frame rate and a second window supporting displayof second image data that refreshes at a second frame rate.Additionally, it is sometimes necessary to display additional image datain another window at the second frame rate or indeed a different framerate. Each window can constitute a plane of image data, the plane beinga collection of all necessary graphical elements for display at aspecific visual level, for example a background, a foreground, or one ofa number of intermediate levels therebetween. Currently, GUIs managedisplay of, for example, video data generated by a dedicated applicationsuch as a media player, on a pixel-by-pixel basis. However, as thenumber of planes of image data increases, current GUIs becomeincreasingly incapable of performing overlays of the planes in real timeusing software. Known GUIs that can support multiple overlays in realtime expend an extensive number of Million Instructions Per Second(MIPS) with associated power consumption. This is undesirable forportable, battery-powered, electronic equipment.

Alternatively, additional hardware is provided to achieve the overlayand such a solution is not always suitable for all image displayscenarios.

One known technique employs to so-called “plane buffers” and apresentation frame buffer for storing resultant image data obtained bycombination of the contents of the two plane buffers. A first planebuffer comprises a number of windows including a window that supportstime-varying image data, for example, interposed between foreground andbackground windows. The window that supports the time-varying image datahas a peripheral border characteristic of a window and a bordered areain which the time-varying image data is to be represented. Thetime-varying image data is stored in a second plane buffer andsuperimposed on the bordered area by hardware by copying the content ofthe first plane buffer to the resultant plane buffer and copying thecontent of the second plane buffer to the presentation plane buffer toachieve combination of the contents of the two plane buffers. However,due to the crude nature of this combination, the time-varying image datadoes not reside correctly relative to the order of the background andforeground windows and so can overlie some foreground windows resultingin the foreground windows being incorrectly obscured by the time-varyingimage data. Additionally, where one of the foreground windows refreshesat a similar frame rate to that of the time-varying image date,competition for “foreground attention” will occur, resulting inflickering as observed by a user of the portable electronic equipment.

Another technique employs three plane buffers. A pair of plane buffersare employed in which a first plane buffer comprises, for example, datacorresponding to a number of windows constituting a background part of aGUI, and a second plane buffer is used to store frames of time-varyingimage data. The contents of the first and second plane buffers arecombined in the conventional manner described above by hardware and thecombined image data stored in a resultant plane buffer. A third planebuffer is used to store windows and other image data constituting aforeground part of the GUI. To achieve a complete combination of imagedata, the content of the third plane buffer is transferred to theresultant plane buffer in order that the image data of the third planebuffer overlies the content of resultant plane buffer where appropriate.

However, the above techniques represent imperfect or partial solutionsto the problem of correct representation of time-varying image data by aGUI. In this respect, due to hardware constraints, many implementationsare limited to handling image data in two planes, i.e. a foregroundplane and a background plane. Where this limitation does not exist,additional programming of the GUI is required in order to supportsplitting of the GUI into a foreground part and a background part andalso manipulation of associated frame buffers. When the hardware of theelectronic equipment is designed to support multiple operating systems,support for foreground/background parts of the GUI is impractical.

Furthermore, many GUIs do not support multiple levels of video planes.Hence, representation of additional, distinct, time-varying image databy the GUI is not always possible. In this respect, for each additionalvideo plane, a new plane buffer has to be provided and supported by theGUI, resulting in consumption of valuable memory resources. Furthermore,use of such techniques to support multiple video planes is notimplemented by all display controller types.

STATEMENT OF INVENTION

According to the present invention, there is provided a method oftransferring image data and an image processing apparatus as set forthin the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

At least one embodiment of the invention will now be described, by wayof example only, with reference to the accompanying drawings, in which:

FIG. 1 is a schematic diagram of an electronic apparatus comprisinghardware to support an embodiment of the invention; and

FIG. 2 is a flow diagram of a method of transferring image dataconstituting the embodiment of the invention.

DESCRIPTION OF PREFERRED EMBODIMENTS

Throughout the following description identical reference numerals willbe used to identify like parts.

Referring to FIG. 1, a portable computing device, for example a PersonalDigital Assistant (PDA) device with a wireless data communicationcapability, such as a so-called smartphone 100, constitutes acombination of a computer and a telecommunications handset.Consequently, the smartphone 100 comprises a processing resource, forexample a processor 102 coupled to one or more input device 104, such asa keypad and/or a touch-screen input device. The processor 102 is alsocoupled to a volatile storage device, for example a Random Access Memory(RAM) 106, and a non-volatile storage device, for example a Read OnlyMemory (ROM) 108.

A data bus 110 is also provided and coupled to the processor 102, thedata bus 110 also being coupled to a video controller 112, an imageprocessor 114, an audio processor 116, and a plug-in storage module,such as a flash memory storage unit 118.

A digital camera unit 115 is coupled to the image processor 114, and aloudspeaker 120 and a microphone 121 are coupled to the audio processor116. An off-chip device, in this example a Liquid Crystal Display (LCD)panel 122, is coupled to the video controller 112.

In order to support wireless communications services, for example acellular telecommunications service, such as a Universal MobileTelecommunications System (UMTS) service, a Radio Frequency (RF) chipset124 is coupled to the processor 102, the RF chipset 124 also beingcoupled to an antenna (not shown).

The above-described hardware constitutes a hardware platform and theskilled person will understand that one or more of the processor 102,the RAM 106, the video controller 112, the image processor 114 and/orthe audio processor 116 can be manufactured as one or more IntegratedCircuit (IC), for example an application processor or a basebandprocessor (not shown), such as the Argon LV processor or the i.MX31processor available from Freescale Semiconductor, Inc. In the presentexample, the i.MX31 processor is used.

The processor 102 of the i.MX31 processor is an Advanced Risc Machines(ARM) design processor and the video controller 112 and image processor114 collectively constitute the Image Processing Unit (IPU) of thei.MX31 processor. An operating system is, of course, run on the hardwareof the smartphone 100 and, in this example, the operating system isLinux.

Whilst the above example of the portable computing device has beendescribed in the context of the smartphone 100, the skilled person willappreciate that other computing devices can be employed. Further, forthe sake of the conciseness and clarity of description, only parts ofthe smartphone 100 necessary for understanding the embodiments hereinare described; the skilled person will, however, appreciate that othertechnical details are associated with the smartphone 100.

In operation (FIG. 2), a GUI software 200, for example QT for Linux,provides a presentation plane 202 comprising a background or “desktop”204, background objects, in this example a number of background windows206, a first intermediate object, in this example a first intermediatewindow 208 and a foreground object 210 relating to the operating system;the purpose of the foreground object 210 is irrelevant for the sake ofthis description.

The presentation plane 202 is stored in a user-interface frame buffer212 constituting a first memory space, and is updated at a frame rateof, in this example, 5 frames per second (fps). The presentation plane202 is achieved by generating the desktop 204, the number of backgroundobjects, in this example background windows 206, the first intermediatewindow 208 and the foreground object 210 in the user-interface framebuffer 212. Although represented graphically in FIG. 2, as one wouldexpect from the IPU working in combination with the display device 122,the desktop 204, the number of background windows 206, the firstintermediate window 208 and the foreground object 210 reside in theuser-interface frame buffer 212 as first image data.

The number of background windows 206 includes a video window 214associated with a video or media player application, constituting asecond intermediate object. A viewfinder applet 215 associated with thevideo player application also generates, using the GUI, a viewfinderwindow 216 that constitutes a third intermediate object. In thisexample, the video player application supports voice and video overInternet Protocol (V2IP) functionality, the video window 214 being usedto display first time-varying images of a third party with which a userof the smartphone 100 is communicating. The viewfinder window 216 isprovided so that the user can see a field of view of the digital cameraunit 115 of the smartphone 100 and hence how images of the user will bepresented to the third party during, for example, a video call. Theviewfinder window 216 of this example overlies, in part, the videowindow 214 and the first intermediate window 208, and the foregroundobject 210 overlies the viewfinder window 216.

In this example, a video decode applet 218 that is part of the videoplayer application is used to generate frames of first video images 220constituting a video plane, that are stored in a first video planebuffer 222 as second, time-varying, image data, the first video planebuffer 222 constituting a second memory space. Likewise, the viewfinderapplet 215 that is also part of the video player application is used togenerate frames of second video images 226, constituting a second videoplane, which are stored in a second video plane buffer 228, constitutinga third memory space, as third, time-varying, image data. In thisexample, both the second and third, time-varying, image data isrefreshed at a rate of 30 fps.

In order to facilitate combination, firstly, of the first video images220 with the content of the user-interface frame buffer 212 and,secondly, of the second video images 226 with the content of theuser-interface frame buffer 212, a masking, or area-reservation, processis employed. In particular, the first video images 220 are to appear inthe video window 214, and the second video images are to appear in theviewfinder window 216.

In this example, first keycolour data, constituting first mask data, isused by the GUI to fill a first reserved, or mask, area 230 bounded bythe video window 214 where at least part of the first video images 220is to be located and visible, i.e. the part of the video window 220 thatis not obscured by foreground or intermediate windows/objects. Likewise,second keycolour data, constituting second mask data, is used by the GUIto fill a second reserved, or mask, area 232 within the viewfinderwindow 216 where at least part of the second video images 226 is to belocated and shown. The first and second keycolours are colours selectedto constitute first and second mask areas to be replaced by the contentof the first video plane buffer 222 and the content of the second videoplane buffer 228, respectively. However, consistent with the concept ofa mask, replacement is to the extent that only parts of the content asdefined by the first and second reserved, or mask, areas 230, 232 aretaken from the first video plane buffer 222 and the second video planebuffer 228 for combination. Consequently, portions of the first videoplane buffer 222 and the second video plane buffer 228 that replace thefirst and second keycolour data corresponding to the first and secondmask areas 230, 232 are defined, when represented graphically, by thepixel coordinates defining the first and second mask areas 230, 232,respectively. In this respect, when the video window 214 is opened bythe GUI, the location of the first mask area 230 defined by the pixelcoordinates associated therewith and the first keycolour data arecommunicated to the IPU by the application associated with the firstkeycolour data, for example the video decode applet 218. Likewise, whenthe GUI opens the viewfinder window 216, the location of the second maskarea 232 defined by the pixel coordinates associated therewith and thesecond keycolour data are communicated to the IPU by the applicationassociated with the second keycolour data, for example the viewfinderapplet 215. Of course, when considered in terms of frame buffers thepixel coordinates are defined by memory or buffer addresses of the videowindow 214 and the viewfinder window 216.

Use of the keycolours by the IPU to implement the first and second maskareas 230, 232 is achieved, in this example, through use of microcodeembedded in the IPU of the i.MX31 processor to support an ability totransfer data from a source memory space to a destination memory space,the source memory space being continuous and the destination memoryspace being discontinuous. This ability is sometimes known as “2D DMA”,the 2D DMA being capable of implementing an overlay technique that takesinto account transparency defined by, for example, either keycolour oralphablending data. This capability is sometimes known as “graphicscombine” functionality.

In particular, in this example, the IPU uses the acquired locations ofthe video window 214 and the viewfinder window 216 to read theuser-interface buffer 212 on a pixel-by-pixel basis using a 2D DMAtransfer process. If a pixel “read” out of the previously identifiedvideo window 214 as used in the 2D DMA transfer process is not of thefirst keycolour, the pixel is transferred to a main frame buffer 236constituting a composite memory space. This process is repeated until apixel of the first keycolour is encountered within the first videowindow 214, i.e. a pixel of the first mask area 230 is encountered. Whena pixel of the first keycolour is encountered in the user-interfacebuffer 212 corresponding to the interior of the video window 214, the 2DDMA transfer process implemented results in a corresponding pixel fromthe first video plane buffer 222 being retrieved and transferred to themain frame buffer 236 in place of the keycolour pixel encountered. Inthis respect, the pixel retrieved from the first video plane buffer 222corresponds to a same position as the pixel of the first keycolour whenrepresented graphically, i.e. the coordinates of the pixel retrievedfrom the first video plane buffer 222 corresponds to the coordinates ofthe keycolour pixel encountered. Hence, a masking operation is achieved.The above masking operation is repeated in respect of the video window214 for all keycoloured pixels encountered in the user-interface buffer212 as well as non-keycoloured pixels. This constitutes a first combinestep 234. However, when pixels of the second keycolour are encounteredin the viewfinder window 216, the 2D DMA transfer process results inaccess to the second video plane buffer 228, because the secondkeycolour corresponds to the second mask area 232 in respect of thecontent of the viewfinder window 216. As in the case of pixels of thefirst keycolour and the first mask area 230, where a pixel of the secondkeycolour is encountered within the viewfinder window 216 using the 2DDMA transfer process, a correspondingly located, when representedgraphically, pixel from the second video plane buffer 228 is transferredto the main frame buffer 236 in place of the pixel of the secondkeycolour. Again, the coordinates of the pixel retrieved from the secondvideo plane buffer 222 corresponds to the coordinates of the keycolourpixel encountered. This masking operation is repeated in respect of theviewfinder window 216 for all keycoloured pixels and non-keycolouredpixels encountered in the user-interface buffer 212. This constitutes asecond combine step 235. The main frame buffer 236 therefore contains aresultant combination of the user-interface frame buffer 212, the firstvideo plane buffer 222 and the second video plane buffer 228 asconstrained by the first and second mask areas 230, 232. The first andsecond combine steps 234, 235 are, in this example, performedseparately, but can be performed substantially contemporaneously forreasons of improved performance. However, separate performance of thefirst and second combine steps can be advantageous where performance of,for example, the second combine step 235 does not have to be performedas often as, for example, the first combine step 234 due to the framerate of the second image data 226 being less than the frame rate of thefirst image data 220.

Thereafter, the content of the main frame buffer 236 is used by thevideo controller 112 to represent the content of the main frame buffer236 graphically via the display device 122. Any suitable known techniquecan be employed. In this example, the suitable technique employs anAsynchronous Display Controller (ADC), but a Synchronous DisplayController (SDC) can be used. In order to mitigate flicker, any suitabledouble buffer or, using the user-interface frame buffer 212, triplebuffer technique known in the art can be employed.

Although the first and second reserved, or mask, areas 230, 232 havebeen formed in the above-described example using keycolour pixels, thefirst and/or second reserved, or mask, areas 230, 232 can be identifiedusing local alpha blending or global alpha blending properties ofpixels. In this respect, instead of the 2D DMA identifying pixels of oneor more mask area using keycolour parameters, an alphablending parameterof each pixel can be analysed to identify pixels defining the one ormore reserved areas. For example, a pixel having 100% transparency canbe used to signify a pixel of a mask area. The ability to perform DMAbased upon alphablending parameters is possible when using the i.MX31processor.

If desirable, one or more intermediate buffers can be employed to storedata temporarily as part of the masking operation. 2D DMA can thereforebe performed simply to transfer data to the one or more intermediatebuffers, and keycolour and/or alphablending analysis of mask areas canbe preformed subsequently. Once masking operations are complete 2D DMAtransfer processes can be used again simply to transfer processed imagedata to the main frame buffer 236.

In order to reduce net processing overhead and hence save power, thefirst video plane buffer 222 can be monitored in order to detect changesto the first video images 220, any detected change being used to triggerexecution of the first combine step 234. The same approach can be takenin relation to changes to the second video plane buffer 228 andexecution of the second combine step 235.

It is thus possible to provide image processing apparatus and a methodof transferring image data that is not constrained to a maximum numberof planes of time-varying image data that can be displayed by auser-interface. Further, a window containing time-varying image datadoes not have to be uniform, for example a quadrilateral, and canpossess non-right angled sides, for example a curved side, whenoverlapping another window. Additionally, relative positions of windows(and their contents), when represented graphically, are preserved andblocks of image data associated with different refresh rates can berepresented contemporaneously. The method can be implemented exclusivelyin hardware, if desired. Hence, software process serialisation can beavoided and no specific synchronisation has to be performed by software.

The method and apparatus are neither operating system nor user-interfacespecific. Likewise, the display device type is independent of the methodand apparatus. The use of additional buffers to store mask data is notrequired. Likewise, intermediate time-varying data, for example video,buffers are not required. Furthermore, due to the ability to implementthe method in hardware, the MIPS overhead and hence power consumptionrequired to combine the time-varying image data with the user-interfaceis reduced. Indeed, only the main frame buffer has to be refreshedwithout generation of multiple foreground, intermediate and backgroundplanes. The refresh of the user-interface buffer does not impact uponthe relative positioning of the windows. Of course, the above advantagesare exemplary, and these or other advantages may be achieved by theinvention. Further, the skilled person will appreciate that not alladvantages stated above are necessarily achieved by embodimentsdescribed herein.

Alternative embodiments of the invention can be implemented as acomputer program product for use with a computer system, the computerprogram product being, for example, a series of computer instructionsstored on a tangible data recording medium, such as a diskette, CD-ROM,ROM, or fixed disk, or embodied in a computer data signal, the signalbeing transmitted over a tangible medium or a wireless medium, forexample, microwave or infrared. The series of computer instructions canconstitute all or part of the functionality described above, and canalso be stored in any memory device, volatile or non-volatile, such assemiconductor, magnetic, optical or other memory device.

1. A method of transferring image data to a composite memory space foroutput by a display device, the method comprising: providing first imagedata in a first memory space, the first image data having a first framerate associated therewith; incorporating mask data into the first imagedata, the mask data defining a reserved output area; transferring atleast part of the first image data and at least part of second imagedata to the composite memory space, the second image data residing in asecond memory space and having a second frame rate associated therewith;wherein the mask data is used by a masking process in relation to thesecond image data in order to provide the at least part of the secondimage data substantially in place of the mask data such that, whenoutput, the at least part of the second image data occupies the reservedoutput area.
 2. A method as claimed in claim 1, wherein the compositememory space is a main frame buffer for the display device.
 3. A methodas claimed in claim 1, wherein the first image data constitutes apresentation plane.
 4. A method as claimed in claim 1, wherein the firstimage data corresponds to a graphical user interface.
 5. A method asclaimed in claim 1, wherein the first image data defines, when output, aplurality of display objects. 6.-13. (canceled)
 14. A method as claimedin claim 1, wherein the first memory space is a first frame bufferand/or the second memory space is a second frame buffer.
 15. A method asclaimed in claim 1, wherein the first frame rate is different from thesecond frame rate. 16.-19. (canceled)
 20. A method as claimed in claim1, wherein the at least part of the second image data is, when output,disposed amongst the output of the first image data.
 21. A method asclaimed in claim 1, wherein the mask data defines a display locationamongst the first image data, when output.
 22. A method as claimed inclaim 1, wherein the mask data is used to by the masking process inrelation to the second image data so that the at least part of thesecond image data is selected when transferred to the composite memoryspace. 23.-29. (canceled)
 30. A method as claimed in claim 1, furthercomprising: employing a DMA transfer process to provide the maskingprocess in relation to the second image data and transfer the at leastpart of the second image data to the composite memory space.
 31. Amethod as claimed in claim 1, further comprising the step of: monitoringthe at least part of the second image data; and wherein the at leastpart of the second image data is provided substantially in place of themask data in response to detection of a change in the at least part ofthe second image data.
 32. A computer program product including codeportions for performing, when run on a programmable apparatus,operations for transferring image data to a composite memory space foroutput by a display device, the operations comprising: providing firstimage data in a first memory space, the first image data having a firstframe rate associated therewith; incorporating mask data into the firstimage data, the mask data defining a reserved output area; transferringat least part of the first image data and at least part of second imagedata to the composite memory space, the second image data residing in asecond memory space and having a second frame rate associated therewith;wherein the mask data is used by a masking process in relation to thesecond image data in order to provide the at least part of the secondimage data substantially in place of the mask data such that, whenoutput, the at least part of the second image data occupies the reservedoutput area.
 33. An image processing apparatus, the apparatuscomprising: a processing resource arranged to transfer, when in use,image data to a composite buffer for output by a display device; a firstbuffer comprising, when in use, first image data, the first image datahaving a first frame rate associated therewith; wherein the processingresource supports a masking process and is arranged to incorporate maskdata into the first image data, the mask data defining a reserved outputarea; and wherein the processing resource supports data transfer and isarranged to transfer at least part of the first image data and at leastpart of second image data to the composite buffer, the second image dataresiding in a second buffer and having a second frame rate associatedtherewith; wherein the mask data is used by the masking process inrelation to the second image data in order to provide the at least partof the second image data substantially in place of the mask data suchthat, when output, the at least part of the second image data occupiesthe reserved output area.
 34. A method as claimed in claim 2, whereinthe first image data constitutes a presentation plane.
 35. A method asclaimed in claim 2, wherein the first frame rate is different from thesecond frame rate.
 36. A method as claimed in claim 15, wherein the atleast part of the second image data is, when output, disposed amongstthe output of the first image data.
 37. A method as claimed in claim 21,wherein the mask data is used to by the masking process in relation tothe second image data so that the at least part of the second image datais selected when transferred to the composite memory space.
 38. A methodas claimed in claim 3, further comprising: employing a DMA transferprocess to provide the masking process in relation to the second imagedata and transfer the at least part of the second image data to thecomposite memory space.
 39. A method as claimed in claim 15, furthercomprising the step of: monitoring the at least part of the second imagedata; and wherein the at least part of the second image data is providedsubstantially in place of the mask data in response to detection of achange in the at least part of the second image data.